Implement auto-verification for plural-native workflows by michaeljguarino · Pull Request #3921 · pluralsh/console

michaeljguarino · 2026-07-28T03:34:03Z

If we know a stack run is spawned by a workbench, auto-verify once it was successful
for services, leverage the new queued prompts to enqueue 15m in the future, when it's almost certainly reconciled.

Test Plan

Test environment: https://console.your-env.onplural.sh/

Checklist

I have added a meaningful title and summary to convey the impact of this PR to a user.
If required, I have updated the Plural documentation accordingly.
I have added tests to cover my changes.
I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).

Plural Flow: console

soffi-ai · 2026-07-28T03:34:21Z

Soffi AI Summary

This PR introduces an auto-verification system for plural-native AI workbench workflows, closing the feedback loop between AI-driven operations and their outcomes without requiring manual intervention.

Core motivation: When a workbench AI agent triggers a stack run or service deployment, there was no mechanism to automatically verify whether the operation succeeded. This PR adds a "queued prompt" concept — a deferred prompt enqueued for future execution — that allows the system to schedule a follow-up verification check once a stack run completes or a service has had time to reconcile.

Key changes:

Queued Prompts schema & pipeline — A new QueuedPrompt Ecto schema (lib/console/schema/queued_prompts.ex) is introduced with a dequeableAt timestamp, enabling time-deferred prompt scheduling. A Broadway-based pipeline (lib/console/pipelines/ai/queued_prompt/) processes these prompts when they become eligible. A database migration adds the required fields.
Stack run auto-verification — When a workbench spawns a stack run and it succeeds, the pubsub recursion handler (lib/console/deployments/pubsub/recurse.ex) now automatically triggers a verification prompt rather than requiring manual follow-up.
Service deployment verification — The service inspection tool (lib/console/ai/tools/workbench/infrastructure/service_inspect.ex) and its prompt template now enqueue a delayed verification prompt (~15 minutes out) using queued prompts, giving services time to reconcile before the AI checks their status. The service prompt template is updated to inform the agent of this behavior.
PR follow-up enqueueing — A new GraphQL mutation enqueueWorkbenchPrFollowup allows enqueueing a verification prompt tied to a PR URL, supporting post-merge verification workflows.
GraphQL & OpenAPI surface — New mutations (createQueuedPrompt, enqueueWorkbenchPrFollowup) and types are added to the Absinthe GraphQL schema, OpenAPI spec, and the auto-generated Go client (go/client/), making queued prompts accessible from both the frontend and Go operator.
Workbench mentions — A new Console.AI.Workbench.Mentions module handles @mention style references within workbench prompts, enabling richer prompt context.
RBAC — Queued prompt operations are properly gated behind RBAC policies.

Commits

Commit	Summary
`9e1bf6c`	Introduces the `QueuedPrompt` Ecto schema with `dequeableAt` scheduling support, CRUD operations in the workbenches context, a Broadway-based producer/pipeline for dequeuing eligible prompts, database migration, RBAC policies, and GraphQL/OpenAPI surface for queued prompts.
`8fa0528`	Adds auto-generated Go client mutations (`CreateQueuedPrompt`, `EnqueueWorkbenchPrFollowup`) and the corresponding GraphQL fragment/types in `go/client/`, wiring the new queued prompt API into the Go operator client interface.
`cad73f9`	Implements auto-verification logic: stack runs spawned by a workbench now trigger an automatic verification prompt on success (via pubsub recursion); service deployments enqueue a delayed (~15m) queued prompt to verify reconciliation; adds the `enqueueWorkbenchPrFollowup` mutation for post-merge PR verification; adds a `Mentions` module for workbench prompt context; updates service inspection tool and prompt templates to reflect deferred verification behavior.

Deploy in Soffi

Updated: 2026-07-28 16:39 UTC

greptile-apps · 2026-07-28T03:36:41Z

Greptile Summary

Implements automatic verification for successful workbench-originated stack runs.

Records the stack run created when a merged stack pull request is polled.
Sends a verification prompt to the associated workbench after successful completion.
Adds the pull-request-to-stack-run schema relationship and migration.
Updates the bundled deployment agent to v0.6.54.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
lib/console/deployments/pubsub/recurse.ex	Records merged pull requests' generated stack runs and invokes workbench verification on successful stack-queue runs.
lib/console/deployments/workbenches.ex	Adds the successful-stack-run lookup and verification-message creation flow.
lib/console/schema/pull_request.ex	Adds the stack-run association, query helper, and changeset field.
lib/console/schema/workbench_job.ex	Adds a query for locating a workbench job through a pull request's stack run.
priv/repo/migrations/20260727224601_add_wb_verification_fields.exs	Adds the nullable indexed stack_run_id foreign key to pull requests.
test/console/deployments/pubsub/recurse_test.exs	Covers recording generated stack runs and creating verification activity after successful completion.

_{Reviews (2): Last reviewed commit: "Implement auto-verification for plural-n..." | Re-trigger Greptile}

michaeljguarino · 2026-07-28T03:58:33Z

@greptileai

michaeljguarino · 2026-07-28T15:57:46Z

plural deploy this

plural-copilot

Deploy PR is here: https://github.com/pluralsh/plrl-up-demos/pull/2389

* If we know a stack run is spawned by a workbench, auto-verify once it was successful * figure out appropriate service equivalent.

michaeljguarino · 2026-07-28T16:32:46Z

plural deploy this

plural-copilot

Created deploy PR: https://github.com/pluralsh/plrl-up-demos/pull/2390

michaeljguarino · 2026-07-28T16:52:45Z

plural deploy this

plural-copilot

Created deploy PR: https://github.com/pluralsh/plrl-up-demos/pull/2391

michaeljguarino requested a review from a team July 28, 2026 03:34

michaeljguarino added the enhancement New feature or request label Jul 28, 2026

greptile-apps Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread lib/console/deployments/pubsub/recurse.ex

Comment thread lib/console/deployments/workbenches.ex

michaeljguarino force-pushed the verification-loops branch 4 times, most recently from 462e9c3 to 6cb4bec Compare July 28, 2026 15:44

plural-copilot Bot reviewed Jul 28, 2026

View reviewed changes

michaeljguarino added 2 commits July 28, 2026 12:14

Implement auto-verification for plural-native workflows

cad73f9

* If we know a stack run is spawned by a workbench, auto-verify once it was successful * figure out appropriate service equivalent.

add queued prompts schema and CRUD

9e1bf6c

michaeljguarino force-pushed the verification-loops branch from 4b802df to c105d9e Compare July 28, 2026 16:15

plural-copilot Bot reviewed Jul 28, 2026

View reviewed changes

go gql queued prompt mutations

8fa0528

michaeljguarino force-pushed the verification-loops branch from c105d9e to 8fa0528 Compare July 28, 2026 16:39

plural-copilot Bot reviewed Jul 28, 2026

View reviewed changes

JohnBlackwell approved these changes Jul 28, 2026

View reviewed changes

michaeljguarino merged commit eccd673 into master Jul 28, 2026
48 checks passed

michaeljguarino deleted the verification-loops branch July 28, 2026 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement auto-verification for plural-native workflows - #3921

Implement auto-verification for plural-native workflows#3921
michaeljguarino merged 3 commits into
masterfrom
verification-loops

michaeljguarino commented Jul 28, 2026 •

edited

Loading

Uh oh!

soffi-ai Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 28, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

michaeljguarino commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Plan

Checklist

Uh oh!

soffi-ai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Soffi AI Summary

Commits

Uh oh!

greptile-apps Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

michaeljguarino commented Jul 28, 2026

Uh oh!

plural-copilot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

michaeljguarino commented Jul 28, 2026 •

edited

Loading

soffi-ai Bot commented Jul 28, 2026 •

edited

Loading

greptile-apps Bot commented Jul 28, 2026 •

edited

Loading